Hey @chaokunyang
Have a review and let me know the changes

hey @ayush00git, looked through this and the main issue i see is in DeserializeFromReader —
it calls ResetWithReader at the start of every call:

func (f *Fory) DeserializeFromReader(r io.Reader, v any) error {
    defer f.resetReadState()
    f.readCtx.buffer.ResetWithReader(r, 0) // this wipes the prefetch window every time

so if fill() reads ahead past the first object boundary (which it will), those bytes
are gone on the next call. sequential decode from one stream is broken:

for {
    var msg Msg
    f.DeserializeFromReader(conn, &msg) // bytes after first object get thrown away
}

if you look at how he handles this for c++/python — the Buffer is constructed
from the stream once and passed to each deserialize call directly. the buffer holds
state across calls, it's never reset between objects. the python test
test_stream_deserialize_multiple_objects_from_single_stream shows this exactly —
same reader buffer passed to multiple fory.deserialize() calls.

the go version probably needs something similar — a stream reader type that owns the
buffer and gets reused across deserializations rather than resetting on each call.

Happy to discuss if I'm misreading the flow here

Hiii @Zakir032002
Thanks for noticing this, exactly this is a bug in the implementation from my side. yes the call would clear any prefetched data from the ByteBuffer making the sequential reads from the stream impossible, also it was clearing the typemetadata as well. thanks for mentioning this, i'll look at the c++ python implementation to correct the deserializer.

hey @ayush00git , one more thing — ReadBinary and ReadBytes return a direct slice into
b.data:

v := b.data[b.readerIndex : b.readerIndex+length]
return v

the problem is fill() compacts the buffer in-place:

copy(b.data, b.data[b.readerIndex:])

so if someone reads a []byte field and holds onto that slice, then the next
read triggers a fill() — the compaction just overwrote the bytes they're
still holding. no error, no panic, just wrong data.

in stream mode you probably want to copy before returning instead of aliasing:

if b.reader != nil {
    result := make([]byte, length)
    copy(result, b.data[b.readerIndex:b.readerIndex+length])
    b.readerIndex += length
    return result
}

in-memory path stays as is.

also noticed — ReadVarUint32Small7 only does fill(1) for the first byte, but if that byte has 0x80 set it falls through to continueReadVarUint32 which isn't touched in this PR. so in stream mode, if a multi-byte varint straddles a chunk boundary, the continuation bytes may not be in the buffer yet — you either get a BufferOutOfBoundError or silently read the wrong bytes depending on what's sitting at that position in the buffer.

easiest fix is probably just routing the multi-byte case through readVarUint32Slow since that's already stream-aware after your changes. or adding fill(1) guards inside continueReadVarUint32 directly, either works.

Happy to discuss if I'm misreading the flow here

Hey @Zakir032002
Sorry i'm a bit busy with my exams, as i get free, i'll review the comments

Hii @Zakir032002
Thanks for pointing out the flows,

• The ReadFromDeserializer and returning a direct slice into the data stream are wrongly implemented by me, thanks for suggesting the chnages to fix them as well.

But i think you misunderstood the ReadVarUint32Small7. We already have a check condition -

if len(b.data)-readIdx >= 5 {

}

If we are near a chunk boundary (less than 5 bytes remaining in the buffer), the execution completely skips continueReadVarUint32 and jumps straight to readVaruint36Slow. I don't think this part need any changes

I've added the StreamReader which now creates a copy slice during desrialization to preserve the data between sequential desrialization calls. DeserializeFromReader only is there if the user wants to deserialize a single struct and don't want a stream overhead for that.

@chaokunyang
the api design now matches with c++, is there any other modification ?

I added shrink_input_stream in #3453, could you add similiar feature in this PR? And I renamed StreamReader to InputStream in #3449, please also rename related API too. the StreamReader is not that clear

Some Design suggestions:

• Keep fast-path varint checks purely remaining >= N; do not call fill(N) in the predicate.
• Never pass nil error sinks to fill in decode methods that promise bounds errors.
• Make DeserializeFromReader either strictly stateless (always reset buffer) or explicitly stateful and align it with InputStream semantics.
• Remove InputStream.reader unless it is actually needed (or use it for invariants/validation).

And please run benchmarks/go for current branch and pache/main branch, and paste the result here

Benchmark: apache/fory/main vs this branch (feat/go-deserialization)

apache/fory/main:

goos: linux
goarch: amd64
cpu: 12th Gen Intel(R) Core(TM) i5-1235U
BenchmarkFory_Struct_Serialize-12              34537162     110.4 ns/op      0 B/op    0 allocs/op
BenchmarkFory_Struct_Deserialize-12            19927696     179.6 ns/op     32 B/op    1 allocs/op
BenchmarkFory_StructList_Serialize-12           3048019    1176 ns/op        0 B/op    0 allocs/op
BenchmarkFory_StructList_Deserialize-12         1583568    2456 ns/op      688 B/op    3 allocs/op
BenchmarkFory_Sample_Serialize-12              14559171     247.5 ns/op      0 B/op    0 allocs/op
BenchmarkFory_Sample_Deserialize-12             3322015    1158 ns/op      676 B/op    9 allocs/op
BenchmarkFory_SampleList_Serialize-12            919764    3815 ns/op        0 B/op    0 allocs/op
BenchmarkFory_SampleList_Deserialize-12          240294   20650 ns/op    13952 B/op  163 allocs/op
BenchmarkFory_MediaContent_Serialize-12         7116196     458.0 ns/op      0 B/op    0 allocs/op
BenchmarkFory_MediaContent_Deserialize-12       2302726    1496 ns/op      656 B/op   13 allocs/op
BenchmarkFory_MediaContentList_Serialize-12      532838    7471 ns/op        0 B/op    0 allocs/op
BenchmarkFory_MediaContentList_Deserialize-12    143080   30065 ns/op    13040 B/op  243 allocs/op

this branch:

goos: linux
goarch: amd64
cpu: 12th Gen Intel(R) Core(TM) i5-1235U
BenchmarkFory_Struct_Serialize-12              32973771     121.9 ns/op      0 B/op    0 allocs/op
BenchmarkFory_Struct_Deserialize-12            27100645     163.6 ns/op     32 B/op    1 allocs/op
BenchmarkFory_StructList_Serialize-12           3265543    1103 ns/op        0 B/op    0 allocs/op
BenchmarkFory_StructList_Deserialize-12         1268991    2679 ns/op      688 B/op    3 allocs/op
BenchmarkFory_Sample_Serialize-12              12737720     258.3 ns/op      0 B/op    0 allocs/op
BenchmarkFory_Sample_Deserialize-12             3287950    1123 ns/op      676 B/op    9 allocs/op
BenchmarkFory_SampleList_Serialize-12            995695    3745 ns/op        0 B/op    0 allocs/op
BenchmarkFory_SampleList_Deserialize-12          231688   18862 ns/op    13952 B/op  163 allocs/op
BenchmarkFory_MediaContent_Serialize-12         6446667     542.3 ns/op      0 B/op    0 allocs/op
BenchmarkFory_MediaContent_Deserialize-12       2616025    1222 ns/op      656 B/op   13 allocs/op
BenchmarkFory_MediaContentList_Serialize-12      414025    8415 ns/op        0 B/op    0 allocs/op
BenchmarkFory_MediaContentList_Deserialize-12    172894   28446 ns/op    13040 B/op  243 allocs/op

@chaokunyang
I'm done with the changes. pasted the benchmarks as well. have a look now